<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<head>
   <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
   <meta name="Author" content="Build">
   <meta name="GENERATOR" content="Microsoft FrontPage 4.0">
   <title>Eclipse Platform Core RFC 0009 - Improved Build Story</title>
</head>
<body>

<h1>
Improved Build Story</h1>
<i><font size=-1>Last revised 17:15 Thursday February 14, 2002</font></i>
<p>This proposal covers a series of additions and changes to Eclipse to improve the build story
and provider better Ant integration.
<h2>
Ant</h2>

<h3>
Running existing Ant scripts inside Eclipse</h3>
Part of the perceived value of Eclipse's Ant support is that the Ant scripts
written outside Eclipse can be run without modification within Eclipse.
This is crucial if Eclipse is to attract open source developers working
with existing code bases and Ant scripts.
<p>This is not well supported in Eclipse 1.0. The problem is that Eclipse
Ant assumes tight integration with Eclipse, with all that entails. Some
of things that don't work well:
<ul>
<li>
Ant scripts that use optional predefined Ant tasks, like javac. (The JDK
tools.jar needs to be on Ant's runtime classpath.)</li>

<li>
Ant scripts that use specialized Ant tasks. (There needs to be a way to
get other jar libraries on Ant's runtime classpath.)</li>

<li>
Ant scripts that call System.exit. (These cause Eclipse to exit.)</li>
<li>
  <p ALIGN="LEFT">Ant scripts that call javac don't cooperate with incremental
  builders. Since auto building is on by default, users can easily end up in a
  situation where both autobuild and Ant script generates class files, but in
  different locations. For running the program, the auto built class files are
  used.</li>
<li>
  <p ALIGN="LEFT">Ant scripts that call javac generate compiler error messages
  that just up in the Ant console. There is no way to navigate from the error
  message to the source file.</li>
</ul>
<p>
As a general goal, we need to find ways to make Eclipse Ant support these
developers. Here are some general properties that the solution should have:</p>
<ul>
<li>
Running Ant as a separate OS process should have the same semantics as
running Ant as part of the running Eclipse, except when Eclipse-specific
tasks are invoked.&nbsp;</li>

<li>
Communication from Eclipse proper to Ant should be via straightforward
Ant command line parameters that users could plausibly furnish themselves.
This makes it easy to call existing existing Ant scripts in the way they
are already familiar with.</li>

<li>
Eclipse Ant should work with Ant buildfiles wherever they are, whether
outside or inside the workspace.</li>

<li>
The user should control the choice of working directory that Ant runs in.</li>
</ul>
If it is easy to run inside Eclipse Ant scripts written for a non-Eclipse
environment, the converse follows automatically: developers working in
Eclipse should be able to write and run scripts that they intend to also
use outside Eclipse. Obviously, these scripts should avoid Eclipse-specific
Ant tasks, which are not available outside Eclipse. While Eclipse Ant should
provide added value for Eclipse users, we should avoid doing things that
gratuitously lock users into Eclipse.&nbsp;
<p>Using Ant's &quot;if&quot; or &quot;unless&quot; clause makes it quite easy to conditionalize
scripts so that they work both inside and outside Eclipse. All that is required
is a single well known variable that is given some value when Ant is run inside
Eclipse.</p>
<p>The proposed changes to Eclipse Ant are as follows:
<ul>
  <li>Run Ant using the same VM as Eclipse, but use a special class loader for
    running Ant, so that the classpath available to Ant can be controlled.</li>
  <li>Since running an Ant script is a long-running operation, it will happen in
    a separate ModalContext thread. <font FACE="Times New Roman">This allows
    that the UI to update and the user to press a Cancel button. However, it
    implies that </font> Eclipse-specific Ant tasks will have to be written in
    such a way that they do not deadlock the system.</li>
  <li>If feasible, install a security manager to prevent an Ant task from using System.exit
    to kill Eclipse.</li>
  <li>Use code from the Ant distribution as is; remove existing hacks.</li>
  <li>Provide an execution environment consistent with Ant run outside Eclipse.</li>
  <li>Use the Ant logger class feature to collect output for the Ant Console..</li>
  <li>Investigate how to provide progress and cancellation while Ant is running
    a script.</li>
</ul>
<h3>
The "refresh" problem</h3>
The following problem becomes painfully apparent when an Ant script written
to run outside Eclipse is run from inside Eclipse. The norm for an Ant
script is to process some files and create, overwrite, or delete others.
For an Eclipse user, there is a good chance that the files in question
are somewhere in the workspace. The problem is that the Eclipse workspace
has no way of knowing where in the workspace the affected files might be.
Without further information, a refresh local on the entire workspace would
be needed to ensure that the in-memory resource tree accurately reflects
the local file system. This is a very bad situation. Refreshing the entire
workspace is time-consuming (at least 0.2 ms per file). But if it is not
done automatically, the user would have to worry about it and handle it
manually. The user may have more knowledge that would allow them to refresh
more selectively --- but it would still be their responsiblity to do it.
<p>By its nature, the refresh operation modifies the resource tree. Modifying
the resource tree triggers resource change event notifications. And if
auto-build is turned on, resource tree modifications also triggers incremental
project builders. Thus even a simple thing like refresh can have far-reaching
consequences.
<p>Here are three options for dealing with this:
<ul>
<li>
<b>Include calls to RefreshLocal in Ant scripts.</b></li>

The RefreshLocal task is Eclipse-specific and would only be available
when the script in run from Eclipse. This is the most precise option since
script authors are in the best position to pin point files in need of refreshing.
<li>
<b>Selectively refresh workspace after running Ant scripts.</b></li>

For each Ant command, allow the user to specify which part of the workspace
needs to be refreshed: none, entire workspace, or specific project (more
generally, a resource working set, if there were such a thing). This makes
it easy for the user to ensure that the refresh happens automatically and
to control how much of the workspace is scanned.
<li>
<b>Autonomous "smart" refresh.</b></li>

A background task scans the local files system for files that are out
of sync with the workspace and compiles a list. Through the UI, the user
controls when the files on this list are refreshed in the workspace. This
is a very general option, with no particular connection with Ant scripts.
Its drawbacks are that it requires a somewhat pervasive visibility in the
UI (like a global toolbar action), and that it is not a programmatic solution.
</ul>
<p>We plan to support the first and second options, and pursue the third option as a separate work
item.</p>

<h3>
Running Ant Scripts from the Workbench</h3>
In Eclipse 1.0, Ant has a very simple UI:
<ul>
<li>
Ant preference page to set verbosity options and change colors or Ant Console</li>

<li>
Toolbar button for running Ant with a history of previously launched scripts.</li>

<li>
Context menu item "Run Ant..." appears on *.xml resources.</li>

<li>
Optional global action set with pulldown action for running an Ant script.</li>

<li>
Ant Console view, which shows the output in a structured way, grouped by task.</li>
<li>
  <p ALIGN="LEFT">Errors in Ant scripts are shown with problem markers.</li>
</ul>
The main user requirements are:
<ul>
<li>
It should be easy for a user to configure and run an Ant script. This includes
selecting the Ant buildfile, selecting one of the Ant targets, and supplying any command line arguments that
might be required. Some of the users will be unfamiliar with Ant and would
not able to read an Ant script.</li>

<li>
  <p ALIGN="LEFT">It should be convenient for the user to select which Ant
  target to run from among the choices in an Ant script.</li>

<li>
It should be easy for a user to re-run an Ant script that they have run
recently.</li>
</ul>
The "Run..." action of the Windows Start menu is a simple UI that covers
both use cases, and serves as a good model.
<p>The proposal is to add a new item to the "Workbench" menu, called "Tool
Script&quot;, which would bring up a Tool Scripts dialog. The Tool Scripts
dialog would allow the user to specify the path of the Ant buildfile and
command line arguments; the last N such commands would be remembered in
a pulldown list.
<p>[There are a couple of places in the Eclipse UI where similar kinds of things
are done (e.g., Debug and Run pulldowns, search). The UI guidelines should
recommend a style of presentation for these kinds of things; this would help
fight a proliferation of different presentations, or similar presentations that
are not similar enough.]
<p>Additional properties:
<ul>
<li>
The dialog should show the full command line so that users can use cut/copy/paste
to transfer commands between themselves. All information which Eclipse
communicates with the Ant script should be disclosed.</li>

<li>
Allow the user to indicate the portion of the workspace to refresh after running
the script: none, or specified container (working set?).</li>

<li>
Allow anything that is runnable on that OS in addition to Ant buildfiles;
e.g., on Windows, let the user select any *.exe or *.bat file.</li>

<li>
<font color="#000000">Provide general support for external commands, and
augmented support for Ant (prompt for target, structured output, verbosity
levels, etc.).</font></li>

<li>
Allow Ant buildfiles (or other runnables) to be outside of workspace as
well as inside a project.</li>

<li>
Paths for buildfiles outside the workspace should ordinarily be specified
with absolute paths in the local file system.</li>

<li>
Browse button to navigate to Ant buildfile (or other runnable).</li>

<li>
Eclipse provides small set of documented parameters that can be used in
the command line. These parameters get macro expanded each time the command
is run. Parameters include: ${project-<i>projectName</i>} - the absolute local
file system path of the named project's root directory; other suggestions
are the path of the currently selected resource or file, current date or
time, location of "workspace", name of currently selected project.</li>

<li>
It should be possible for plug-ins to extend the set of available parameters. For example, JDT
might provide parameters like ${project-output-folder-<i>projectName</i>} so
that the user can readily direct a script to generate class files into a Java
project's output folder.</li>

<li>
Paths for buildfiles inside project should ordinarily be specified in project-relative
terms; e.g., as ${project-myproject}/scripts/ant1.xml where "myproject" is the
name of a workspace project.</li>

<li>
Consider allowing tool scripts to show up in a submenu of the Tool Scripts menu
item so that we can map an accelerator to the actions.</li>

<li>
The Tool Script action is in a global action set that could be added to
a perspective's tool bar (either programmatically, or by the user via Perspective
-> Customize).</li>

</ul>
<p>
Implementation details:</p>
<ul>
<li>
Command lines in the Tool Scripts dialog are saved as UI state.</li>

<li>
By default, command lines would not be considered to be shared metadata.
If there were a mechanism for sharing metadata between users (another work
item), there might be merit in letting the user specify that certain commands
were to be shared.</li>
</ul>
Other issues to be resolved:
<ul>

<li>
Ant Console view presents the output from running an Ant script to the user.</li>

<ul>
<li>
Could the Ant Console be made to show output from running other runnables?</li>

<li>
How does the Ant Console view come to the user's attention? Would this view be
pre-configured into the standard perspectives? Or would the user have to ask for
it?</li>

<li>
To be consistent with the Debug console, the Ant Console should be made visible when output is written
to it.</li>
</ul>

<li>
Support back navigation from the Ant Console to other views. Without the ability to do this (at least in some limited cases), we don't have an integration story.
We should use a pluggable approach similar to that used for the Open Selection
action in the JDT Console.
<ul>
  <li>N.B. The CDT commander View lets you specify patterns for the line and define how to interpret a line. This is used for error navigation.</li>
</ul>
</li>
</ul>

<h2>
Build Story</h2>

<p>
By the Eclipse build story we mean the mechanisms underlying the
&quot;Build&quot; and &quot;Rebuild all&quot; actions that appear in the generic
workbench UI.</p>

<h3>
Running Ant Scripts as Builders</h3>
A ordinary user would be unlikely to care which incremental builders a
project has, or what order they run in. The user just hits "Build". These
details are for the plug-in developer who provides the incremental project
builders and the project natures which configure them.
<p>However, we would like a way to give advanced user the ability to extend
what happens when the user hits "Build".
<p>The proposal is to allow the user to add additional Ant commands to
the project's list of builders.
<ul>
<li>
Build commands in a project build spec would be extended to include things
other than incremental project builders.</li>

<li>
Would allow arbitrary commands as per Tools Script dialog.</li>

<li>
Could be couched as running a special incremental project builder with
all info stuffed in arguments. (There is a bug in current implementation
that precludes multiple build commands from using the same builder id.)</li>

<li>
Project properties page for viewing and extended a project's incremental
project builders.</li>

<li>
User can add or remove extra commands before or after any step in build
sequence.</li>

<li>
User cannot add, remove, change, rearrange, or disable builders. This is a responsibility of
the plug-in/project nature.</li>

<li>
Additional parameter variable available for command line: ${thisProject}
- the absolute local file system path of this project's root directory;
${buildType}
- &quot;full&quot;, &quot;incremental&quot;, or &quot;auto&quot;.</li>

<li>
Multiple commands per step so that it is possible to configure one for incremental
build and one for
full build.</li>

<li>
Check box to indicate whether autobuilds should be included with other
incremental builds.</li>

<li>
A way to handle scripts which are failing (a "stop on error" checkbox to fail
the Build operation when this happens?).</li>

<li>
Allow the user to indicate the portion of the workspace to refresh after running
the script: none, or specified container (working set?).</li>

<li>
Project build spec is persistent project metadata, shared via the project's
description.</li>

<li>
Extra build steps are therefore automatically shared.</li>

<li>
Individual builders provide a pre-project preference setting to allow the user to
disable it if they can forsee users wanting to (and being permitted to) replace
it. For example, it should be possible to disable the Java builder for a project
so that
the user can run an Ant script that calls a javac instead.</li>

<li>
Core should treat missing scripts (and other problems) like it does missing/failing
builders.</li>

<li>
Core must catch reentrant case of IncrementalBuild task being invoked from an
Ant script being run as a builder.</li>

</ul>

<h3>
Ordering projects</h3>
When several projects are involved in a build, the individual projects get
built, one at a time, in a certain order. Unless explicitly overridden by the
user, the order of project is the default ordering of projects in the workspace
as computed by Core using a topological sort based on project references (if
project X refers to project Y then build Y before building X).
<p>At the very least, we can improve the user experience by taking the following
steps :</p>
<ul>
  <li>Users should be discouraged from overriding the default project order
    computed by Core. This is because only the default project order gets
    updated automatically when project references change or additional projects
    are introduced into the workspace.</li>
  <li>Cycles in project references are detected and quietly broken when Core
    computes the default project order, often leading to miserable orders which
    do not work and only frustrate and confuse the user. When Core is unable to
    compute a surefire default project order, the user must be informed. The
    user should be kept aware of the problem as long as it persists; otherwise,
    the user is liable to forget about the problem and release to a team
    repository a project with screwed up project references, thereby propagating
    the problem to other developer workspaces.</li>
</ul>

<h3>
Ordering builders within a project</h3>
Particular incremental project builders are typically inserted in the project's
build specification when it is configured to have a project nature (the
code for the project nature does this). However, each project can be configured
to have a number of project natures, in no particular order. At present,
this mechanism is quite weak. How can a plug-in add a builder I after some
other builder J when the project does not have a J? It is reminiscent of
each program adding a line to the beginning (or end) of autoexec.bat when
it gets installed, with no guarantee that things will work properly if
you don't install the programs in the expected order.
<p>Note that this is not the kind of question that we want to involve the
user in. This is an internal matter for the plug-ins or project natures
that are configuring a project. A reasonable solution therefore requires
some scheme or mechanism so that plug-in can definitely resolve builder
ordering issues.
<p>In a separate work item, we are proposing to impose constraints on the
order project natures get added to (or removed from) a project. For instance,
if project nature B was known to have nature A as a prerequiste, then Core
could arrange that a project to be configured with both A and B would get
A first, then B. This would mean that the code for project nature B would
be able to assume that A had already added its builders to the project.
Consequently, A could determine where to add its builders to the project
taking into account the positions of B's builders (assuming that A knows
the ids of the builders that B adds).
<br>&nbsp;
</body>
</html>
